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(54) Image forming apparatus 

(57) An image forming process for easily making an 
image drawing (ortho-image) in an actual place while 
confirming it in real time. Measuring of control points (at 
least three points) is performed (S110), and preparation 
is made for forming an ortho-image based on image 
data thereof and survey data. Where photographing is 
carried out from a plurality of directions, additional 
image measuring is performed (S140). Then, ortho- 
image formation is performed on a PC (S1 60). Here, if a 
formed ortho-image is not what is desired, ortho-image 
correction is performed (S180). After a satisfactory 
image is obtained (S200). the process moves to a next 
area to be measured, and the same operation is 
repeated. On the other hand, unless a satisfactory 
image is obtained, additional image measuring (SI 40) 
is performed again. 
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Description 

BACKGROUND OF THE INVENTION 

[0001 ] The present invention relates to an image forming apparatus, an image forming method and a computer-read- 
able storage medium having an image forming program. More particularly, the present invention relates to a portable 
and simple image measuring technology utilized for surveying and making a drawing in a land surveying field, which 
uses digital processing by digital photogrammetry and a personal computer. 

[0002] Specifically, the present invention relates to an image forming technology for forming a digital orthogonally pro- 
jected image (ortho-image) from one to a plurality of images to be superposed on the drawing when a drawing of a site 
to be measured is made. The present invention relates to an image forming technology which enables any operator to 
easily make a drawing of an orthogonally projected image by using a survey instrument and images so as to stick an 
ortho-image to the drawing, and also enables a cEgrtal orthogonally projected image to be generated and rectified such 
that the situation of a site to be measured can be understood in detail. An object of the present invention is to provide 
an image which enables stable stereo analysis to be performed even for a stereo-image. 

[0003] The present invention can also be applied to, for example, an image obtained by photographing a wide range 
divided into small images having overlapping areas with one another. 

[0004] According to a prior art. a drawing obtained by surveying in a site to be measured has been made by using 
paper and a pencil or the like which are typically used in plane table surveying. In recent years, a line drawing of a site 
to be measured has been made by using a survey instrument and a portable computer in combination which are typi- 
cally used in pen-based mapping system. 

[0005] Fig. 18 is a view illustrating conventional stereo-image measuring. Usually, as shown in the drawing, the three- 
dimensional coordinates are obtained by photographing two or more overlapped images based on the principle of tri- 
angulation by stereo-image photographing. If a target range is wide, a plurality of images are photographed. For exam- 
ple, in even a simple site, photographing of ten to twenty images is usually necessary. In this case, unless orientation 
(i.e.. calculating a camera position, inclination or the like) can be performed for each stereo pair (model), a stereo-image 
cannot be rectified. Three-dimensional measuring cannot be performed, either. 

SUMMARY OF THE INVENTION 

[0006] In the conventional case, however, a drawing was expressed only by a line image even if it was made in the 
actual place, and it was impossible to sufficiently understand the situation of the site. Thus, the situation of the site was 
photographed by a camera. However, since photographing was performed from an oblique direction whereas a direc- 
tion of a requested drawing was vertical, a photographed image had no direct connection with the drawing. As a result, 
it was necessary tor an operator to understand the situation of the site by comparing the drawing with the image. This 
practice was inconvenient, and understanding of the site was difficult 

[0007] A range to be photographed with one image was limited. Even in the case of photographing a plurality of 
images, because correlation and continuity among the respective photographs are lacking, this led to the difficulty of 
obtaining consistency among the photographs, and comparison with the drawing was complicated and difficult. If a 
highly accurate ortho-image was required, lens distortion of the camera was an obstacle. Conventionally, it was impos- 
sible to produce a highly accurate ortho-image by a camera which has no data on lens distortion. 
[0008] In the case of conventional land surveying, for surveying a site to be photographed and maWng a drawing for 
the site, it was necessary to survey an enormous number of points and perform three-dimensional measuring. Thus, 
much man power and labor had to be expended. On the other hand, if a dose and terrestrial photogrammetry is used, 
indirect survey three-dimensional data can be obtained only by photographing. But since only a camera using a film was 
available conventionally, developing and scanning took time (e.g., two to three days), and several days were also 
expended for analyzing work. Unless processing up to analysis was carried out after film development, there was no 
knowing whether photographed image data was analyzable or not, and whether stable, sure and highly reliable analysis 
was possible or not. As a result, conventionally, this method has not been used so often because the necessity to again 
perform the work such as photographing may happen. 

[0009] With popularized and increased use of digital cameras in recent years, digital close and terrestrial photogram- 
metry can now be performed by a digital camera which is different from a conventional film or analog camera. Use of 
this technology eliminates labor and time for film development and scanning. Since analysis is performed by a compu- 
ter, the process from photographing to analyzing can be finished within one to two days. 

[001 0] However, after acquisition of photographic data, it was necessary to bring the photographic data to the place 
of an analyzing device (computer) for analysis and measuring. Thus, photographing improper for analysis might result 
in the necessity to again perform the work such as photographing, and stable and sure analysis was not always carried 
out.. 
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[001 1 J For performing analysis and measuring by images, if control point/orienting point arrangement, a photographic 
range, photographic overlapping or the like was not proper, stable and highly reliable analysis was impossible. Conse- 
quently, unstable results occurred, and even analysis itself was impossible. 

[0012] The present invention was made with the foregoing points in mind. The object of the present invention is to 
provide an image forming apparatus for, when a drawing of the situation of a site to be measured is made, easily making 
a drawing of the site from an orthogonally projected image (ortho-image) so as to facilitate understanding of the situa- 
tion. The object of the present invention is to quickly and easily generate/rectify an image in the site to be measured 
without any storage omission or mistakes, to understand the situation of the site on the spot, and to easily make an 
image drawing (ortho-image) in real time while confirming the same. The object of the present invention is to perform 
image complementary measuring only by simple photographing and surveying of several points with a survey instru- 
ment, to simultaneously obtain an image drawing for facilitating understanding of the situation of the site, to interpolate 
portions not photographed or hard to make out. and to produce an ortho-image of a high resolution and wide range. 
[001 3] Another object of the present invention is to facilitate understanding of a situation from a plurality of images, 
even if the situation is hard to grasp, for example, some parts are difficult to see or a far part becomes rough with one 
image, and produce a highly accurate ortho-image. The object of the present invention is to integrate a plurality of 
images by repeating simple work and obtain a wide-ranging ortho-image. Further, the object of the present invention is 
to form a highly accurate and high-quality ortho-image at a high speed by photographing a plurality of images over- 
lapped with adjacent areas. 

[0014] The still another object of the present invention is to correct lens distortion (camera calibration correction) by 
simple measuring, and simultaneously form a highly accurate ortho-image even by a camera having no lens distortion 
data. Further, the ot^ect of the present invention is to form a good quality ortho-image of a necessary place with 
required accuracy by using various survey instruments to rectify the ortho-image while measuring the three-dimen- 
sional coordinates. 

[001 5] According to the present invention, the following operations and confirmation can be performed in a site to be 
photographed: 

(1) confirmation of a measuring range (e.g.. overlapping); 

(2) generation and confirmation of an ortho-image obtained by integrating plurality of images; 

(3) confirmation of the entire measuring4>hotographing range and a range to be drawn. 

(4) confirmation of entire arrangement of control points/orienting points; and 

(5) confirmation of orientation, and confirmation of a model formed for stereo method analysis. 

[001 6] In accordance with first solving means of the present invention, there is provided an image forming apparatus, 
which comprises: 

a control point measuring section for measuring a centrally projected image having a plurality of control points, and 
obtaining the image coordinates for said control points; 

a coordinate transformation parameter calculating section for obtaining a transformation parameter for correlating, 
based on said image coordinates for said control points obtained by said control point measuring section and the 
three-dimensional coordinates for actually measured control points, said image coordinates with said three-dimen- 
sional coordinates; 

an orthogonally projected image (ortho-image) forming section for forming an orthogonally projected image from 
said centrally projected image based on said transformation parameter obtained by said coordinate transformation 
parameter calculating section; and 

an ortho-image correcting section for correcting the image coordinates obtained by said ortho-image forming sec- 
tion based on the three-dimensional coordinates for actually measured additional points, and then performing cor- 
rection of said orthogonally projected image. 

[0017] In accordance with second solving means of the present invention, there are provided an image forming 
method, an image forming apparatus and a computer-readable storage medium having an image forming program. In 
this case, the image forming method comprises: 

an image input function, to which a plurality of images including control points or orienting points overlapped with 
one another are inputted; 

a storage function for previously storing ground coordinate values of control points or orienting points; 
an orthogonally projected image (ortho-image) forming function for forming an ortho-image from plurality of images 
inputted by said image input function based on image coordinate values, alternatively photographic coordinate 
value, and said ground coordinate values of the control points or the orienting points; and 
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i^^^m Kv/ caiH nrtho-imaae forming function, 
a determining function for determining, based on the ortho-image formed by &*a ° or 
necessity of re-photographing, and necessity of chang.ng a photograph.ng pos,t.on. aiterna y ^ 
an orienting point position. 

RRIPF DESCRIPTION OF THE DRAWINGS 
[0018] 

F,g. 1 isacor^Hution^^^ 

Fig 2 is a flow chart of an image forming process according to a ^J^JJJ 8 

F; , 3 is an explanatory view showing a case where a widei range Qf tne m invention . 
Fig 4 is a flow chart of an image forming process according to a second emDoaime v 
F.g. 5 is a flow chart of on-line control point measuring. , measuring section. 

Fia 6 is a flow chart of automatic control point measuring performed by a control pu 
F« 7 is an explanatory view showing an example of control po.nt arrangement 
J£ 8 Is a i low chart of coordinate transformation parameter calculation process. 
Fig. 9 is a flow chart of additional image measuring. 
Fig. 10 is a flow chart of an ortho-image forming process. 

Fig. 1 1 is a flow chart of ortho-image correction process. imnrooe r imaae area 

B 12 is aflow chart of delaying a control ^iS^ «~ 

Fia 1 3 is an explanatory view showing an example of a control point snonage * r ^ 

Rg. 4 S S ^chart oUlaying an image shortage place o^operplace 
Fia 15 is an explanatory view showing displaying of an image shortage ^ P 
fS 4 J an explanatory view showing the camera coordinates and the mode, coord.nates. 
25 Fia 1 7 is a flow chart of off-line processing. ^nirwaDhina 
32. 18 is an explanatory view showing conventional image measuring and photographing. 

PF TA" P" DESCRIPTION OF THF P RFFFRR EP FMFOPIMENTS 
, .0019] Now. the preferred embodiments of .he present invert will be desc^^ 

drawings. 

A Constitution and operation oi image forming apparatus 
,0020, ng-lfcacons^on^ 

ratus comprises a control unit 10. a storage section 2. an ^°^ ^ c ^f ^ unH 1 0 furthermore 
section 7. an image Input section 4. i^*"" & andan ^^^ meter gating section 102. an 
includes a control point measunng section 101. a ^'^^7^^ an additional image measuring section 
ortho-image forming section 103. an ortho-image correcting section ^ a ™*" a pC) 
1 05 These section^ (functions) can be realized by. "^JJ^ZZZv the like. The control point 
I0 021 ] The storage section 2 previously stores coordinates. The input/output 

data is stored in. for example, the three-d.mens.onal ^^fj^^^Sus devices. The three-dimensional 
interface 3 is composed of a common bus or the like, and addrtiona. measuring points 

coordinates input section 7 obtains the ^'^^^'^ps ~S Sation. and so on. or a three<Jimen- 
by various survey instruments. e.g.. a gtaba. ^™***«" 'ib^S£*£*»* or tridimensional image 
sional coordinates measuring device. The image input *J%Lto d images including control points 

from, for example, a digital camera. From the image input secton 4. one or a puira ny 

are inputted. _ .. nicniav a olasma display or the like, and performs 

[0022] The display section 5 is provided with a CRT . . hquid ^^mage formed by the ortho- 

L^fimensiona. a thredimensional displaying^ ^^^^^O.^e^s^SB^s- 
image forming section 103. the orthc-image correcting ***™^**l™Z\n . an image added by the additional 
plays a control point position inputted by the control pcxrrt ^XlnWerTvarious image data and resufts of deter- 
fmage measuring section 105 or the like. The taP*^"*" ^ fr ^ ^ee<limensiona. coordinates input 
mination. alternatively the three-dimensional | OOortta«lwdata _or *e IM ™™ ^ 6> va(ious ^output 

device such as a survey instrument or the like wrth oth* .unteJJ* •« '^JJJ a memory or ^ like), a floppy 
devices, for example, an optical disk device, a card re«xd p^ed. The input/output section 6 

disk, a keyboard, a pen. a mouse, a terminal and a C 0 -^**™'"!^" of the display section 5 by a pointing 
gives instructions regarding control points and various positions on the screen oi tne a p 



35 



40 



45 



SO 



4 



EP 0 964 223 A1 



device such as a mouse, a write pen or the like. 

[0023] The control point measuring section 101 measures a centrally projected image having a plurality of imprinted 
control points, and obtains the image coordinates for the control points. The control point measuring section 101 meas- 
ures, by using, as the three-dimensional coordinate input section 7, a survey instrument for measuring a distance and 
s an angle from an already-known point, the three-dimensional coordinates for the control points based on the obtained 
coordinates, and stores the same in the storage section 2 as required. Depending on the control points, the control point 
measuring section 1 01 can automatically obtain each centrally projected image coordinate value corresponding to each 
control point for which the three-dimensional coordinates has been obtained. 

[0024] The coordinate transformation parameter calculating section 102 obtains a transformation parameter. This 

w transformation parameter is used for transforming the image coordinates obtained by the control point measuring sec- 
tion 101 into the photographic coordinates having an original point set as a principal point (a point where a line passing 
through a lens center and orthogonal to the screen intersects the image), and correlating, based on the image coordi- 
nates and the three-dimensional coordinates for the control points, which have been obtained by the control point 
measuring section 101, the image coordinates with the three-dimensional coordinates. Also, as described later, the 

is coordinate transformation parameter calculating section 102 can calculate, by satisfying a collinear condition for central 
prelection and using an image input section where a principal distance is known, a transformation parameter based on 
at least three control points, for which a ground coordinate system is known. Further, the coordinate transformation 
parameter calculating section 102 can correct lens distortion based on lens distortion data of the image input section 
or actually measured data of a plurality of control points. 

20 [0025] The ortho-image forming section (ortho-image forming function) 1 03 transforms the three-dimensional coordi- 
nates into the image coordinates based on the transformation parameter obtained by the coordinate transformation 
parameter calculating section 102, and forms an orthogonally projected image. For forming an orthogonally projected 
image by sticking a plurality of images, the ortho-image forming section 103 can select each image obtained based on 
a distance from each image to a control point or a distance from each image to a measuring point 

25 [0026] Also, the ortho-image forming section 103 forms an ortho-image from image coordinate values (or photo- 
graphic coordinate values) of the control points/orienting points and ground coordinate values based on a plurality of 
stereo images inputted by the image input section 4. The ortho-image forming section 103 can include a function of dis- 
criminately displaying a non-overlapped area/non-photographed area determined by the ortho-image correcting section 
104 or a shortageflailed positioning of the control points/orienting points on the display screen of the ortho-image. Fur- 

30 ther, the ortho-image forming section 103 can include a function of executing orientation such as successive orienta- 
tion, relative orientation or the like based on a plurality of stereo images. 

[0027] The ortho-image correcting section 104 corrects the image coordinates obtained by the ortho-image forming 
section 103 based on the three-dimensional coordinates for actually measured additional points, and performs correc- 
tion of the orthogonally projected image. Also, the ortho-image correcting section 104 can form, if a plurality of centrally 

35 projected images are obtained, an orthogonally projected image by combining images selected according to a proper 
standard, e.g., images of places having specified/lower reduction scales or having relatively small reduction scales, 
alternatively images close to the measuring point or the control point, on priority basis. Further, the ortho-image cor- 
recting section 104 can display on the display section 5, if a plurality of centrally projected images are obtained, an 
image shortage area or an improper area which results from formation of the orthogonally projected image performed 

40 by the ortho-image forming section 103. 

[0028] A determining function included in the ortho-image correcting section 104 determines, based on the ortho- 
image formed by the ortho-image forming section 103. necessity of changing a photographing position and necessity 
of changing positions of the control points/orienting points. This determining function includes a function of extracting a 
non-overlapped area, the area being not overlapped by at least two stereo images in the ortho-image. A determining 

<5 section 12 includes a function of extracting a non-photographed area which is not covered with by stereo images in a 
measuring target area. Also, the determining function includes a function of determining a shortage/failed positioning 
of control points/orienting points included in a portion overlapped by at least two stereo images in the ortho-image. Fur- 
ther, the determining function includes a function of outputting, if results of orientation performed by the ortho-image 
forming section 1 1 are not converged, an instruction for changing setting of the control points/orienting points to both or 

so either of the ortho-image forming section 103 and the additional image measuring section 105. 

[0029] A correction data forming function included in the ortho-image correcting section 104 forms data for re-photo- 
graphing if the determining function detects a non-overlapped area/non-photographed area or a shortage/failed posi- 
tioning of the control points/orienting points in the ortho-image. This data for re-photographing is necessary for solving 
the detected failure, and for this data, for example, data regarding a photographing position, data regarding a photo- 

55 graphing range, data regarding selection of control points/orienting points, data for specifying control points/orienting 
points to be rearranged or the like can be used. Such data is fed back to the ortho-image forming section 103. 
[0030] The additional image measuring section 105 measures other centrally projected images so as to include the 
control points measured by the control point measuring section 101 , and calculates transformation parameters for these 
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centrally projected images by using the coordinate transformation parameter calculating section 1 02. 
10031] Next, an overall operation will be summarized. 

[0032] Fig. 2 is a flow chart of an image forming process according to a first embodiment of the present invention The 
drawing shows a case where an image is formed based on one or more images and threeKJimensional coordinate val- 
ues by a survey instrument. An image forming process carried out in the control unit 10 is described hereinbelow bv 
referring to Fig. 2. 7 

[0033] The process shown in Fig. 2 has two courses of processing: on-line processing performed in an actual place 
and off-line processing for performing only control point measuring in the actual place and bringing back remaining jobs 
to be finished in an office or the like. In the on-line processing, different from the case of the off-line processing control 
point measuring, subsequent image forming, displaying, and so on. are carried out in the actual place. 
[0034] In the image forming process, first measuring of control points (three points or more) is performed (step S1 1 0) 
and preparation is made for forming an ortho-image based on image data thereof and survey data. Furthermore if pho- 
tographing is earned out from a plurality of directions, additional image measuring is performed (step S140) For per- 
forming analysis only with one image, the additional image measuring (step S140) is unnecessary Then, for example 
on a PC. ortho-image formation is performed (step S160). Here, if a formed ortho-image is not what is desired ortho- 
image correction is performed (step S1 SO). In this ortho-image correction work, determination is made as to whether a 
satisfactory image is obtained or not (step S200). IfOK", then the process moves to a next area to be measured and 
the same operation is repeated from the control point measuring (step S1 10). On the other hand, if no satisfactory 
image is obtained in the ortho-image correction work (step S180), then the additional image measuring (step S140) is 
carried out again, and the foregoing operations are repeated until a satisfactory image is obtained. 
[0035] The additional image measuring (step S140) is performed, for example if some parts are hard to see when 
photographing is carried out from one direction, if an image of a far part becomes rough even on one image and thus 
sufficient accuracy is not provided, or the like. In other words, if an ortho-image is formed by performing further photo- 
graphing from another direction for compensating for an already inputted image or yet another direction where there is 
a shortage, a place hard to see or a place of less accuracy will be selected/synthesized from a camera image of another 
direction, and thereby a good quality ortho-image will be formed. 

[0036] Fig. 3 is a view illustrating a case where a wide range is measured. If a range to be measured is wide a meas- 
uring range may be decided in accordance with necessary accuracy, and the image forming process in Fig. 2 may be 
repeated sequentially in respective areas of measuring ranges 1. 2. 3. • • • and so on. of Fig. 3. Necessary accuracy 
depends on. for example, a distance from an object, a size of one CCD pixel or the like. Thus, by dividing a wide range 
for processing, an ortho-image can be easily formed no matter how wide a range is. 

[0037] Fig. 4 is a flow chart of an image forming process according to a second embodiment of the present invention 
The process shown is image measuring processing for performing stable measuring by stereo images. Image measur- 
ing processing is described hereinbelow by referring to Fig. 4. 

[0038] First, control points/orienting points are set, and measured data is set by a survey instrument (step S1 100) 
The control points have been accurately measured on the basis of a ground coordinate system (absolute coordinate 
system) (X. Y, Z) beforehand by the survey instrument. H the orienting points are arranged uniformly (discretely) on a 
photographic image, it is not necessary to measure positions thereof beforehand. Needless to say, the orienting points 
can be also used as position-measured control points. The ground coordinate system (X, Y, 2) is a three-dimensional 
orthogonal coordinate system where real space of a model has been set 

[0039] Then, photographing is performed such that each image can sufficiently overlap another, and the control 
points/onenting points, for example at least six points or more, can overlap one another (step S1200). Here, as an 
example, each image has at least six or more control points/orienting points, however, the proper number thereof can 
be used as occasion demands. For example, if bundle adjustment or the like is used, the number of control points for 
measuring can be reduced to three. Even if orienting points are used instead of control points, the necessary number 
of targets is about six. Alternatively, four points are enough if orienting points can be assumed that they are arranged 
on a plane. 

[0040] After the photographing, various set values necessary for image processing, e.g.. both or either of an ortho- 
image forming range and resolution, and so on, are set onto the portable computer (a PC) (step S1300). Then the con- 
so trol points/orienting points of each image are measured on a PC (step S1400). Here, by the ortho-image forming func- 
tion, orientation and ortho-image formation are automatically carried out according to a program (step S1500) The 
specific processing will be described in detail later. 

[0041 ] After the calculation by the ortho-image forming function, the results of orientation and an ortho-image are dis- 
played on the display screen (step S1600). Then, based on the results of step S1600. each check item is confirmed for 
55 the measured image (step Si 700). 

[0042] The display function displays arrangement of all and each image by the ortho-image. Furthermore, the insuf- 
ficient results of orientation are discriminately displayed based on various bits of display information by which the check 
items are confirmed. Regarding the check items, if the resufts of orientation are insufficient (e.g., targetted accuracy is 
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not achieved, no convergence occurs or the like), improper selection or arrangement of the control points/orienting 
points is determined. A failure of an entire measuring range such as existence of a non-photographed area caused by 
omission of photographing or the like is confirmed by the formed ortho-image. If a non-overlapped area exists because 
of an image overlapping failure, propriety of an overlapped portion is confirmed. If there is such an overlapping failure, 

s by extracting only an overlapped part between adjacent images in the formed ortho-image and displaying the same, a 
gap is formed in a joined part between images on the ortho-image, and thus immediate determination can be made. 
[0043] In addition to the confirmation of the display screen by a measuring operator, the check items can be automat- 
ically confirmed by the control function (especially, the determining function). If the results of orientation do not con- 
verge as in the case of failed arrangement or failed positioning of the control points, or if successive orientation is not 

10 normally carried out the results of orientation containing errors are outputted in the step S1 600 is outputted, and a fail- 
ure such as an improper image is discriminate^ displayed. For example, the control function can allow the display func- 
tion to automatically display the fact that the results of orientation fail to achieve the targetted accuracy or fail to 
converge by an identifier, a flag, and can also allow the display function to discriminately display a proper range by color 
or patterns. Even in the case where a non-photographed area or a non-overlapped area exists, the area can be property 

is recognized by the control function and using an image processing technology. Specifically, for example, a vacant place 
of an image is detected, and by identifying an area/position thereof, the above failures can be determined. 
[0044] If necessity of correction is decided based on the results of checking, then a decision is made as to necessity 
of determining, because of a shortage, a positioning failure of the control points/orienting points, arrangement thereof 
again for re-photographing (step S1720). 

20 [0045] Here, for example, H rearrangement of the control points/orienting points is not necessary because only an 
image photographing position is not proper, as in the case where a non-photographed area or a non-overlapped area 
exists, data for re-photographing is formed by the correction data forming function, and photographing is performed 
again according to this data (step SI 860). Then, the process is resumed again from the step S1400. 
[0046] On the other hand, for example, if the arrangement of the control points/orienting points must be changed as 

25 in the case where the results of orientation are not converged or successive orientation is not carried out normally, rear- 
rangement thereof is examined (step S1740). If there are other control points/orienting points on the image and the 
image coordinates for these points is obtained without re-photographing, selection of the control points/orienting points 
is changed from the already photographed image (step SI 760), and the above calculation is started from the step 
S1500. This processing may be performed automatically. On the other hand, if other proper control points/orienting 

30 points do not exist on the image, arrangement changing is performed by moving the arrangement of the control 
points/orienting points (step S1840). Then, data for re-photographing the image including the rearranged control 
points/orienting points is formed by the correction data forming function, and re-photographing is performed according 
to the formed data (step S1860). Then, the process is executed from the step S1400. 

[0047] The re-photographing processing in the step S1 860, the measuring processing m the step S1 400 may be exe- 
35 cuted only for the image including the control points/orienting points, which is necessary to handle each failure, alter- 
natively for all the images to be photographed. 

[0048] The decision as to necessity of re-photographing based on the results of checking, and the decision as to the 
step to which the process returns when re-photographing is necessary, may be manually made by the operator based 
on the display screen, alternatively made automatically by a PG, otherwise made by proper combination of the manual 
40 operation and a PC. 

[0049] As descrfoed above, when it is decided that any correction is necessary correction is determined based on the 
result of checking in the step S1700. the process is finished with the decision that a desired ortho-image has been 
obtained. 

45 B. Detail of the processing 

[0050] Next, the flowchart of Fig. 2 will be described in detail. 

(1) Control point measuring processing by the control point measuring section 101 (step S1 10). 

50 

[0051] Hereinbetow, control point measuring processing (step S110) by the control point measuring section 101 is 
described in detail. 

(1-1) On-fine measuring 

55 

[0052] First, an example of on-line measuring will be described. Fig. 5 is a flow chart of on-line control point measur- 
ing. In on-line control point measuring for forming an ortho-image on the field of a work site, work is done while forming 
and confirming an ortho-image in an actual place. Thus, according to the on-line control point measuring, non-efficient 
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Ul^Sf)- 88 re : measurin 9 the control P° in <s in the actual place can be avoided, such work being necessary in the case 
of measuring m (S takes or shortage parts found after a photographed image is brought back anS analyzed 

thr c ontrn.^I; i nT! 01 T u,' 66 at ** minimum - are s« in a range to be measured (step S1 12). Furthermore 
SmSnl^ h . Preferably be discriminab| e targets on an image and measured by a survey instrument [three' 

~ Sf° ' , 6 '^ SeCti ° n ?) FOf 016 COntro ' P° ints - for exam P |e - P risn « or reflecting targJTSStalS 
alternatvely those tartgets which return for returning reflecting (measuring) signals can be used in the cSe of a 
prism total stafoir Then, photographing is performed by a digital camera (image input section 4Hstep SIM) jl* oho" 
tographmg may be performed for one image, or the desired number of iniages from a plurality o ^diSctions F Ir exam* 
pie when some parte are hard t0 see if pn 0lographed from one djrectjon 9 wnen jma P J^J£*Z£E^ 
better to perform further photographing from a plurality of directions considered, rt is 

ferrS 1 ^ IHT^? I^V P™ 0 ^* 0 " *V *e digital camera and stored in the storage device of the camera is trans- 
mit nZe^oS " n,t f ima96 ' 0rmin9 and "t*** t0 *° 2 ^ S? 6) The 

Say So^S ? ere Th , ""I™ °° m " ! ° be meaSUred ™ 8 PC " and *• Paraphed image is diSayei in me 
display section S. Here, the targets are collimated by the survey instrument (step S1 1 8) and confined If OK L «Jn3 
pants as «arge te are measured on the image on a PC (step S120). In this case, for example. poSons of' £ So 
pants are instructed by a pen as the input device of the input/output section 6. After the image coordinates <£ ov 
(CCD coordinate system, and an origina. point is set to be a screen end such as the left upper JS5 m^Snffe 
Ti T S T 9 ,nStrUCti0n 15 OUtpUtted to the three-dimensional coordinates input stctionTmeasurS^ 2£ 
^ZT^I J three^imensional coordinates input section 7 (step 8122), and then measured ground JSSSS valui 
ofth . corrtro. pants are sent through the irput/output l/F 3 to the control unit 1 0 and the storage sector^ S2 KM 

mT«r 1 ' * 6 83016 work ,s ^Peated the minimum three points (step S126). 

oSS ST k a ^!r t,C f" 1 '!! P 0 " 1 * ""^"fl ^ control point measuring section 101 will be described 
rTv ie a L JL * automatic "^trol point measuring performed by the control point measuring section 

Fig. 7i s a view showing an example of arrangement of control points easunng section. 

m2?m^ , - COrt E haVin ° reflectin 9 sheets are set (step S501). Attaching the reflecting sheets facilitates auto- 
t as targets to be seen from any directions, and these points can be fixed to. for example apexes of 

rZ/LTiu * " advisable to s « one obviously different in size. Then, as one example, photoaraohino is car- 

^^iSSiX" ^ S5 ° 3) - ACCOrding,y ' situations be detected.^e^ ^ P ToS2^ed 
^tS^^ES!^ 00 ^^ 8S °"* Then> the three<limensiona. coordinates are measured b^tne 

Ts SemSTin aShSS .T" T T the C ° ntr ° l ^ havSnQ the ,ar9est re,lectin 9 8heet - l«*nwfen thereof 
ralSr "n7 I . ^ simultaneously transferred from the survey instrument to the control unit 10 

(Sep SSOOftJ e^!! » r tf* 10 r,ma9e correlation Processing), the image coordinates on the image is detected 

^fS12^SS^^ V* returns ,rom 630,1 of ,he ref lec6na sheets ^ hi9h ,uminance and ^ re 9 is ' 

£^m3ilf?i^f ? ^ 88 lem P |ates (aching patterns) beforehand, position detection is facilitated. For 

Sr the^n^eT^' * ""^ and ^ tem? ' ateS 03,1 Then - *• coordinated 

Sa^blT™^!^. PO ' n,S are COrrela,ed M,h P° inte °» the three^imensional coordinates (step S51 1). For 
SS^Z -rZ?3 ^ ntS and se,tin S one amon 9 them to be large as shown in the drawing, me targe control 
Z!S? m ~ 9 , ^ meS d,scriminable fa y the template, and is correlated based on the attribute measured on the 
Sor^Z COOrd ' nate !- The ° ther ^ P° inte can be correlated based on the three^imensiona. coord^ 
mST "TUTl P ° Si,i0nS there ° f Wen K these P° ints are Photographed from any directions. 

c*f r LthSt rSSSS^lR °. P ° Srt, ° n d6teCti0n " ^ images are Photographed, one with a strobe light and the 
ma« c ^l! t ™? 9 i ^ tek,ng 8 differenc e between these two images, only the targets emerge. Thus, auto- 

dSonI!2 ™^ n9 described datum to the large and small reflecting sheets. Image coordinates 

22^2 7 9 alS ° be Performed by eolorin » or P 3 " 6 " 1 ^ the ref ,ecti "9 6heete - Other than the reflecting 

a^Zf'Srr' °" ^ ima9e CSn 66 Accordingly, there are no limitations placed on the nunSi 

arrangement, shapes, and so on of control point. " 
rnlSJ! NeJrt ternpla,ematchin 9 processing will be described. 

I^ H o( S : 'r^' a, l mat foo r ^' any proper method such as a normalized cross correlation method, a sequential sim- 

^?l 3 intlT^S 0n ? *l* SSDA iS Sh ° wn below - ln expression, a point where a residual R(a.b) becomes small- 
S SSI? ° b ! ain ! d D ^ increasi "9 a Processing speed, for examp.e. in addition o! this expresSn. 
the addition is finished when a value of R(a.b) exceeds a minimum value of a past residual, and calculation isperformed 
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so as to move to next (a,b). 



A/1-1 A/1-1 

m*.b)= X X I Vfc) ( m v n \) ~ ^i,^) I (1) 

m1«0 n 1=0 

Ttnri! t n^y. Template image, 
'(a.bjt" 1 !. n i) : Partis image of target image 
(a,b): Left upper coordinates of template image 
R(a,b): Residual 

(1-2) Coordinate transformation parameter calculation processing by the coordinate transformation parameter calculat- 
ing section 102 (step S300) 

[0064] If the foregoing measuring for the control points is OK, then coordinate transformation parameter calculation 
by the coordinate transformation parameter calculating section 102 is performed (step S300). 
[0065] Fig. 6 is a flow chart of coordinate transformation parameter calculation process. 

[0066] First, since the points measured on a PC are obtained as the image coordinates on a charge coupled device 
(a CCD) such as a digital camera, these coordinates are transformed into a photographic coordinate system (x, y) (step 
S203). The photographic coordinate system (x, y) is the two-dimensional coordinates having an original point set as a 
principal point. On the other hand, the image coordinates are CCD coordinate system (px, py), for example, the two- 
dimensional coordinates having an original point set in the left upper side. 

[0067] The three points measured on the ground coordinate system (X, Y, Z) by measuring of the survey instrument 
and the two points of the image coordinate system (x, y) measured on a PC are substituted for expressions below, and 
each parameter for coordinate transformation (coordinate transformation parameter) is obtained (step S205). Expres- 
sions 2 and 3 are collinear condition expressions where a projection center, an image on a CCD and an object are 
located on a straight line. Accordingly, if there are at least three already-known points, each coordinate transformation 
parameter can be calculated. In this case, however, a principal distance c must be nearly known. If a calibrated camera 
is used, no problem occurs since a principal distance c is known. 

r _ , a 11 (X-X 0 ) + a 12 (/-yoHa 13 (Z-Z 0 ) 

a 31 (X-X 0 )+ag 2 ( r 0 )+a 33 (Z-Z 0 ) v ' 

a 21 (X-X 0 Ka 22 (r-y 0 )+a g 3(Z-Z 0 ) 
y = " c a 31 (X-X 0 )+a 32 (K- Y 0 )+a^[Z-Z 0 ) { } 



Herein, 




x.y 


:lmage coordinates, 


*o.yo 


: Princpal point position, 


X.Y.Z 


: Ground coordinates, 




: Coordinates of projection center, 


c 


: Princpal distance 


a ij 


: Rotation matrix 




: Inclination of camera 



[0068] On the other hand. H a principal distance c is not known, this value can be obtained by measuring coordinate 
values of four points on a plane based on a two-dimensional projective transformation. Also, by using six points, a rela- 
tionship between the image coordinates and the three-dimensional coordinates (target points coordinates) of the object 
can be obtained by an approximate solution based on a three-dimensional projective transformation expression (direct 
linear transformation). 

[0069] HereinbeJowra method by the three-dimensional transformation expression is described. This processing is 
for calculating a transformation parameter for obtaining a pixel position on an ortho-image. Here, the processing is per- 
formed mainly based on known coordinates for the control points/orienting points and the measured image (photo- 
graphic) coordinates. 

[0070] First a basic expression is shown below. 
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_ LiX+L 2 Y+L 3 Z+L 4 
X " L 9 X+L 10 V+L 11 Z+1 W 

L 5 X+L 6 Y+L 7 Z+L 6 



y = 



L s X+L^Y+L„Z+\ 



Herein, 



(x.y) :lmage coordinates. 

(X.Y.Z) :Ground coordinates, 

1-1 to L t1 :Coordinates transformation parameters 

[0071] By calculating the expression 4 by using a least squares method based on data of the control points, coordi- 
nate transformation parameters L, to L t1 for deciding a relationship between the image coordinates (x y) and the 
ground coordinates (X.Y.Z) can be obtained. 

[0072] Further, a coordinate transformation expression is not limited to the foregoing, and any proper coordinate 
transformation expressions can be employed as long as these can correlate the ground coordinates with the imaqe 
coordinates. 

(1-3) Camera calibration 

[0073] The foregoing description has been made of the case where measuring is performed with accuracy for permit- 
ting lens distortion to be ignored or a non-distortion lens is used. However, if there is lens distortion and this distortion 
cannot be ignored for accuracy measuring is carried out with a camera having pre-calculated lens distortion or while 
correcting lens distortion by processing described below. 

[0074] Specifically, in the case of using a camera having no lens distortion data, the lens distortion is corrected by 
measuring six or more control points and obtaining lens distortion of the camera based on the following expressions 5 
to 7. In the case of a camera having pre-calculated lens distortion, coordinate values are obtained while correcting the 
lens distortion by using these expressions from the start. 

x --c a n( X " X o^ a i2(^^oHa 13 (Z-Z 0 ) , 

a 31 (X-X 0 ) + a 32 (y-y 0 > + a33(Z-Z 0 ) +A>r < 5 > 

a 21 (X-X 0 )+a 22 (y- ^Ka ga (Z-Z Q ) 

a 31 (X-X 0 ) + a 32 ( / 0 ) + a 33 (Z-Z 0 ) +Ay < 6 > 



Ax = x 0 + x(k^r 2 + k 2 r 4 ) (j) 



r 2 n=( x 2 + y 2 )/c 2 



k t , \<2 : parameters of radical distortion 



[0075] These expressions are calculated by measuring six or more control points on the ground coordinates and the 
image coordinates and by a successive approximation solution. 

[0076] In addition to the foregoing, by using a direct finear transformation with self-calibration, correction of lens dis- 
tortion can be performed by expressions 8 and 9. In this case, however, at least eight control points are necessary. 

x + Ax = — £ 5 i /o\ 

L 9 X+L l0 V + L n Z + 1 (8) 
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L 5 X+L 6 Y+L 7 Z+L 6 
"/. 9 X+L l0 V + L n Z + 1 



(9) 



[0077] By canceling denominators of the above expressions based on data of the control points and by using the least 
squares method, unknown variables k 1( k 2 . xq and y 0 can be obtained. 

[0078] As described above, lens distortion is obtained by the foregoing expressions, and accordingly, measuring while 
correcting lens distortion can be performed. These expressions are only examples, and calculation may be carried out 
by other expressions. 

[0079] Then, referring again to Fig. 5, after the foregoing coordinate transformation parameters calculation processing 
(step S300) is finished, by using the transformation parameters, a residual between the coordinate transformation val- 
ues and the control points is obtained, and a decision is made as to whether the residual is within a prescribed value or 
not (step S128). If the residual is within the prescribed value, then the process moves to the next processing. On the 
other hand, if the residual is not within the prescribed value, the process returns to the step S1 1 8, and while increasing 
the number of points to be measured, the operations of the steps S11 8 to S128 are performed until the residual comes 
within the prescribed value. 

[0080] Here, the residual can be obtained in the following manner. Specifically, the transformation parameters 
obtained by the expressions 2 and 3 (alternatively the expressions 5 and 6) and the image coordinates (x, y) measured 
for the control points are substituted in expressions 10 and 1 1 described below, and calculation control points X' and Y' 
of the ground coordinate system (X, Y, Z) are calculated. Then, a residual with the actually measured ground coordinate 
values (X. Y) are obtained by an expression 12. If the residual b thus obtained is within the prescribed value, it is OK. 
For the prescribed value, for example, accuracy necessary in an actual place is set. Herein, n'means the number of con- 
trol points. Expressions for the residual should not be limited to the ones described here, but any other expressions can 
be used. 



6 = Jl.(X'-X) 2 +Z{Y'Y) 2 I n (12) 

(2) Additional image measuring processing by the additional Image measuring section 105 (step S140) 

[0081] Next, additional image measuring processing (step S140) will be descrtoed in detail. Fig. 9 is a flow chart of 
additional measuring. If the number of images to be photographed is one. alternatively if an image is formed with one 
image temporarily, this processing can be skipped. Also, if a plurality of images are photographed first in steps S114 
and S1 16. a step of photographing (step Si 42) and a step of inputting an image to a PC (step S144) can be skipped. 
[0082] Photographing of additional images is performed so as to include all the control points set by the control point 
setting (step S1 12) carried out in the control point measuring processing (step Si 10). Then, the images are inputted to 
a PC (step S1 44). the additionally photographed images are displayed on the display section 5. and an additional image 
to be measured is selected (step S146). Then, the control points imprinted on the selected image are measured on a 
PC (step S148). This work is repeated by a number of times equal to the number of control points (step S150). For 
example, rf a portable computer of a pen input type is used, by indicating the control points as targets by a pointing 
device such as a pen, the image coordinates (px, py) (or the photographic coordinates (x. y)) thereof is obtained. Then, 
coordinate transformation parameter calculation like that described above is performed (step S300). 
[0083] Further, for additionally measuring other images, the process returns to the additional image section (step 
S1 46), and this processing is repeated by a number of times equal to the number of images to be photographed. When 
no additional measuring is carried out. the process moves to next step to form an ortho-image (step S1 60). 

(3) Ortho-image formation processing by the ortho-Image forming section 103 (step S1 60) 

[0084] Next, ortho-image formation processing (step S160) will be described in detail. Fig. 10 is a flow chart of an 
image forming processing. 

[0085] First a ground coordinate for each pixel on an ortho-image is calculated (step S207). In this processing, for 
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forming an ortho-image, the image coordinates (x, y) for the ortho-image are transformed into the ground coordinates 
(X, Y, 2). The ground coordinates (X. Y, Z) is calculated by using the transformation parameters previously obtained in 
the step S205 of the coordinate transformation parameter calculation processing (step S300). In other words, the 
ground coordinates (X. Y, Z) corresponding to the image coordinates (x, y) for the ortho-image is provided by the follow- 
5 mg expression. Accordingly, a position for obtaining each pixel on the ortho-image can be obtained. 

10 

■ 2m .^Ltf±d ' (13) 



is Herein, 

(*o. Y 0 ) Position of left upper side of ortho-image in ground coordinate system, 
(AX, AY): Size of one pixel in ground coordinate system (e.g., rrvpixel), 
(x, y) : Image coordinates for ortho-image 
20 (X, Y, Z) :Ground coordinates, 

a, b, c, d : Coefficients of plane equation formed by a plurality of control points interpolating certain image coordi- 
nates (x, y). 

[0086] This time, by using the transformation parameters obtained in step S205, the image coordinates (x, y) corre- 
25 sponcfing to the ground coordinates (X. Y. Z) obtained in step S207 is calculated based on the expressions 2 and 3 or 
the expressions 5 and 6 (step S209). From the obtained image coordinates (x, y), a gray value on the ground coordi- 
nates (X, Y, Z) for the proper image is obtained. This gray value is a pixel gray level in a two-dimensional position (X, Y) 
on the ortho-image. In this way, a gray value of an image to be stuck to a position (X, Y) on the ground coordinates is 
obtained. By performing the foregoing processing for alt the pixels of the ortho-image, image sticking is carried out (steo 
so S211). ^ 

[0087] Hereinbelow, sticking of a plurality of images is described more. 

[0088] If the number of images is not singular but plural, then selection of an image is decided based on a positional 
relationship between the control points and the camera. In other words, a position (Xq. Y 0 , Zq) of a camera for each 
image is calculated based on the expressions 2 and 3. Accordingly, a distance is obtained from values thereof and actu- 
35 ally measured values (X, Y Z) of control point coordinates, a pixel gray value of a closer image is obtained and then the 
image is stuck. Thus, by carrying out image sticking by using an image closer to the control point, image data having 
high resolution can be selected. 



40 



45 



(4) Ortho-image correction processing by the ortho-image correcting section 104 (step S180) 

[0089] Next, ortho-image correction (step SI 80) will be described in detail. Fig. 1 1 is a flow chart of ortho-image cor- 
rection. 

[0090] Ortho-image formation (step S1 60) is performed, checking is made on the formed ortho-image visually or by 
later described processing. If there are measuring mistaken parts or parts which especially need measuring data, 
measuring is performed for these parts the survey instrument (step S184). The three-dimensional coordinates (ground 
coordinates) measured is transferred from the survey instrument to the control unit 10 (step S1 88), the above-described 
ortho-image formation (step Si 60) is automaticaOy executed, immediate displaying of the ortho-image is carried out by 
the display section 5, and then checking is made again. 

[0091] Accordingly, since measuring is performed point by point and processing is performed while confirming a cor- 
rected image in real time, measuring mistakes or measuring omissions can be prevented, and thus measuring can be 
always performed while confirming an image as an end product. 

[0092] Next, a flow illustrating a method of extracting and correcting a control point shortage place or an improper 
place from the formed ortho-image will be described. 

[0093] Fig. 12 shows a flow chart of control point shortage place or improper image place displaying. Fig: 13 is an 
55 explanatory view showing an example of a control point shortage area or an improper image area. 

[0094] First, control point shortage place displaying is executed. The ortho-image formed from one or a plurality of 
images is first displayed (step S601). Then, a range in a measuring area which is intended to be confirmed is specified 
(step S603). Here, whether control points are located in a proper control points arranging area or not is checked, and 
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displayed (step S605). For example, if the number of control points is six, a display is like that shown in Fig. 13. For 
checking control point arrangement, a proper control point range is divided on the ortho-image based on the number of 
control points to be measured, the coordinates of the control points are checked, decision is made as to whether the 
control points are located in the divided sections or not. and a result of the decision is displayed. Alternatively, a frame 
of a proper control point range like that shown in Fig. 13 (in this example, the area is divided into six sections) is dis- 
played. But no limitation is placed on a method for deciding a proper control point range. 

[0095] Subsequently, displaying of an improper image place is executed. First, a gap and a difference of elevation 
between adjacent control points are calculated (step S607). Here, a point of which a calculated value (e.g.. a difference 
of elevation) is equal to or higher than a threshold value is displayed so as to show the fact (step S609). The threshold 
value is calculated from an image reduction scale such that a drastically changed point can be detected from, for exam- 
ple, the number of control points or a gap and a difference of elevation between the control points. 
[0096] Then, as described above, in ortho-image correction (step S180). a control point shortage part and a part in 
the vicinity of the point of which the calculation value is equal to or higher than the threshold value are measured by the 
survey instrument, and then the image is corrected, tf no satisfactory result is obtained even by these procedures, it is 
advisable to execute additional image measuring based on the information regarding these displayed images. 
[0097] By the above measuring and processing, the control point shortage part or the improper part of the ortho- 
image can be corrected. 

[0098] Next, a method for extracting the shortage part or the improper part from the formed ortho-image and display- 
ing the same will be described. Hereinbelow, a method of specifying a shortage/improper part based on image resolu- 
tion will be described. 

[0099] Fig. 14 shows a flow chart of displaying an image shortage part/improper part Fig. 1 5 is an explanatory view 
showing displaying of an image shortage part/improper part. Necessary measuring accuracy is set according to an 
object as it differs from object to object. 

[0100] First, the ortho-image formed from one or a plurality of images is displayed (step S701). Here, cases of per- 
forming photographing from positions A and B are assumed. Then, a range in a measuring area which is intended to 
be confirmed is specified (step S703). Since a camera position (Xq. Y 0 . Zq) has been obtained based on the expres- 
sions 2 and 3. accuracy of one pixel in the measuring confirming area is calculated from this position (step S705). If 
photographing is performed from plural directions, the above calculation is executed for each camera. Then, a place 
where pixel accuracy of each camera does not achieve set accuracy and a range for overlapping are displayed on the 
ortho-image (step S707). Then, additional image measuring is performed from the direction of the displayed place (step 
S709). Herein, a case of photographing additional images from an additional image photographing position C is 
described. 

[0101] Accordingly, an image for compensating for the image resolution (one pixel accuracy) of the image shortage 
place/improper place is obtained and. as a result a satisfactory image can be obtained. The pixel accuracy can be 
determined, for example, in a manner that accuracy is reduced as a ratio of transforming a centrally projected image 
into an ortho-image is increased, or as a distance from an image photographing point to an object having an imprinted 
image is increased. 

(5) Orientation 



[01 02] Next orientation of each model will be described in detail. The orientation is. for exanple, a part of processing 
in step S1500 shown in Fig. 4. 

[01 03] Based on the three-dimensional coordinates of each image obtained in step S1400 shown in Fig. 4, orientation 
can be sequentially performed from the measuring points (e.g., six or more measuring points) such as the control 
points/orienting points imprinted in each model image. The orientation is data used for checking in step S1700. 
[0104] First, as an example, relative orientation is described. For stereo images, positions of left and right cameras 
are respectively obtained by calculation shown below. 

[0105] Fig. 16 is an explanatory view showing the camera coordinates and the model coordinates. 
[0106] In the camera coordinate 6ystem. a lens center (projection center, principal center) is set as an original point, 
and x and y axes are in parallel with x and y axes of the photographic coordinates. If a principal distance is c (oO), then 
a point on a photographic plane is expressed by (x, y. - c). The model coordinate system (X, Y, 2) is a three-dimensional 
coordinate system for defining a stereo image formed from a pair of two stereo photographs, and an original point 
thereof is set in the projection center of the left photograph. 

[0107] First, parameters are calculated under a coplanarity condition expression like the following. 
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(14) 



15 



20 



25 



Herein, 

*02' Y 02. Zo2 
Xi. Y,. Z t 
X2, Y 2 , Z 2 



: Projection center coordinates of left image expressed by model coordinate system, 
: Projection center coordinates of right image expressed by model coordinate system, 
: Image coordinates of left image expressed by model coordinate system, 
: Image coordinates of right image expressed by model coordinate system. 



[0108] Now, as shown in the drawing, an original point of the model coordinate system is set in the projection center 
of the left side, and a line connecting the projection center of the right side is set in the X axis. For a scale, a base line 
length is set to a unit length. Parameters obtained at this time are five rotational angles, i.e.. a rotational angle k, of the 
Z axis and a rotational angle 4i of the Y axis of the left camera, and a rotational angle k 2 of the Z axis, a rotational angle 
4> 2 of the Y axis and a rotational angle of the X axis of the right camera. In this case, it is not necessary to consider 
a rotational angle of the X axis of the left camera as it is zero. 

[0109] Under the above condition, a coplanarity condition expression 15 becomes one like that shown below, and 
each parameter can be obtained by calculating this expression. 



30 



F(k v + v c 2 , + 2 , o 2 ) b 



Y 2 Z 2 



: ^ ! Z 2 - Y 2 Z , 



(15) 



[0110] Herein, between the model coordinates (X, Y, Z) and the camera coordinates (x, y, z), a relational expression 
35 of coordinate transformation like that shown below is established. 
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[011 1] By using the foregoing expressions, unknown parameters are obtained based on the following procedure. 

55 

1 . An initial approximate value is usually set to 0. 

2. A value of differential coefficient is obtained by the expression 7 when the coplanarity condition expression 
(expression 13) is tailor<Jevetoped around the approximate value and made linearization, and then an observation 
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equation is established. 

3. By using the least squares method, a correction amount is obtained for the approximate value. 

4. The approximate value is corrected. 

5. By using the corrected approximate value, operations from 2 to 5 are repeated until results converge. 

[01121 No convergence may occur, for example, as in the case where arrangement of the control points, orienting 
points is improper. In such a case, in the displaying of orientation results in step S1600 of Fig. 4, an improper image is 
discriminated, and then displayed. 

[01 13] If the results of calculating unknown parameters converge, another step, tor example, successive orienting, is 
performed. The successive orientation is for setting an identical coordinate system by unifying inclinations or reduction 
scales of the respective models subjected to orientation and interconnecting a plurality of images. In the successive 
orientation, a pair of stereophotographs are connected to each other by fixing one orientation parameter and operating 
only the other orientation parameter. If this processing is carried out, a connection range represented by the following 
expression is calculated. 

AX j = ( x jr-Xji)/(Zo-Zj.) 
AYj-tYjr-Y^/CZo-Zj,) 
AZ j-(Z jr-Zj^/CZo-Zj,) 

AD j = 7(AX J 2 + AY j 2 ) (17) 

Herein, 

(AXpAj-iYAZp): j-th left model of course coordinate system 
(AXjjAYjjAZj,): j-th right model of course coordinate system 

[01 14] If AZj and ADj are equal to a predetermined value (e.g., 0.0005 (1/2000)) or lower as a result of calculation, it 
is decided that the successive orientation is normally executed. 

[0115] H results of calculation do not converge, or H normal execution of the successive orientation is not decided, 
then an error is outputted in the orientation result display in the step S1600 of Fig. 4, and a failure, such as an improper 
image or the like, is discriminate displayed. 

C.Application 

(1 ) Use of various survey Instruments 

[01 1 6] Next, use of various survey instruments and advantages will be described. 

[0117] For example, if correction work is carried out by the non-prism total station, in the case of an object which 
returns a light (measuring signal), an ortho-images is renewed as occasion demands only by performing measuring for 
the object Accordingly, use of such a 6urvey instruments is extremely effective for facilitating work and providing capa- 
bility of measuring a number of points. 

[0118] Use of another survey instruments is very effective in that if correction processing is carried out by an auto- 
tracking total station, only by walking with a prism in a shortage measuring area, an ortho-image can be formed with 
higher accuracy. 

[01 1 9] Use of the other survey instruments is also greatly effective in that if an area can receive date from satellites, 
only by walking with a GPS in the area, an ortho-image is renewed, and any operator can form an ortho-image while 
easily performing complementary measuring. 

[0120] For obtaining a high-density and highly accurate ortho-image without any failures, it is advisable to perform 
stereo photographing to overlap ranges to be measured, and perform relative orientation an actual place. In this way, 
automatic stereo matching (image correlation processing) is performed from stereo images to obtain high-density 
three-dimensional measuring points and, by forming an image by the ortho-image forming section, high-density and 
highly accurate ortho-image can be obtained. Alternatively, a shortage part, a discontinuous part or the like is manuaDy 
measured on the display while seeing formed stereo images, and by combining the measuring points thereof and then 
forming an ortho-image by the ortho-image forming section, an ortho-image having high luminance can be obtained 
without any failure. 
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( 2) Measuring of wide-ranging area w tne measuring range 1 shown in F.g. 3 

(3 ) Off-line measuring le lor performing on-»ne measuring of control points. 

l01221 The oontro. point ^^^? 7 ^SS^^ tSme^ur'g by the survey instrument 
SSt oH-Une processing will be de ^ ,b ^ F £^ otograp hing and control ^^^e G r the «M- Off-line meas- 

SSl Accordingly, for the control ^JJ^UnQ is carried out ^ Then, additional image 

^Sto-rass^ 

Claims 

calculating section. 

55 3 . A„l«^1or^^^■^^ 1Wi,-, ^ ^asathree^mensiona.coordlnateinput 

.r^4 ha a survey instrument as a uu<= 
said measured control points are measured by a surv y 
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, adv known point amatively by a global portm- 

ing system. 

« *^ riflims 1-4. wherein 

■ 5 Animagetormingapparatusaccordingtooneoitheda bv ^ying a coUinear condition .or 

data of the image input secuon. w 
7 ^wlom-nB^^-^^^^ - ^^ 1 *^ ^edimagebystiddngap.uraHtyot 

IS each image to a measuring point- 
, A, image W*« » <" '* . a.. a***-" 



image Dy saw* w,uw — ° 

or an orierrting point posmon. 
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lapped by the stereo images in said ortho-image. 

,5 w ^. Wn « ( ,™.^ac»« l n»»««o ( * ed a i m S .1.13.«.e,e l n 

15. wi image « » ^~,ranhed area in a measuring target 

MW determining function inciudes a function of extracting a non-photographed 

IS covered with images in satf ortho-.mage. 
1S. ^i^e^ngmetho.a^ordingtooneof^^lt-IS^erein 

sa^imagefo^^ 
17 te Wte r^«^«^»«-^^ 1, - 1ft ^ . tograph 

« ^ imageformingfunctioninciudesafunctionol executing orie^on such as succes.ve o 
SSESJTK like based on a piuralHy d i images, ard ^ ^ ^ ge . an 

19. An image forming apparatus comprising: 

^lor-^wim^^^-^^ 11 - forrrtnam ethod 

according to claim 11. 
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